{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=gray size=4> 2023-03-31 </font> \n",
    "\n",
    "<font color=gray size=4> Python 数据挖掘</font>\n",
    "\n",
    "<font color=gray size=4> 笔记：Kanekikeh </font>\n",
    "\n",
    "<font color=gray size=4> week05:学习爬取“猎聘网”这种动态网站数据</font>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from requests_html import HTMLSession\n",
    "session = HTMLSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 一、首先进入猎聘网随便一个页面 然后打开 开发者工具（F12）查看里面的Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![猎聘数据爬取1](https://pic.imgdb.cn/item/64279464a682492fcc05449d.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 二、进入 Network 以后 进行url的筛选 让传递过来的数据暂时只显示url 然后可以一个一个点击圈起来的url 查看哪个url是从数据库传过来的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![猎聘数据爬取2](https://pic.imgdb.cn/item/64279464a682492fcc0544fb.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 三、经过一个个url里面的Preview查看后 锁定我们想要的url 可以观察一下里面的data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![猎聘数据爬取3](https://pic.imgdb.cn/item/64279465a682492fcc0545be.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 四、然后就像正常的请求数据流程，需要给它一些参数 而这里则是利用payload进行酬载"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![猎聘数据爬取4](https://pic.imgdb.cn/item/64279465a682492fcc05460f.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 五、需要模拟人为打开浏览器的操作 而不是 pyhton打开浏览器的操作，也就是headers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![猎聘数据爬取5](https://pic.imgdb.cn/item/64279465a682492fcc054689.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 六、准备好这些参数就可以开始请求访问数据了，参数分别是：\n",
    "* url\n",
    "* method\n",
    "* payload\n",
    "* headers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "url = 'https://api-c.liepin.com/api/com.liepin.searchfront4c.pc-search-job'\n",
    "payload = {\n",
    "    'data' :{\n",
    "        'mainSearchPcConditionForm':{\n",
    "            'city': \"050020\",\n",
    "            'compId': \"\",\n",
    "            'compKind': \"\",\n",
    "            'compName': \"\",\n",
    "            'compScale': \"\",\n",
    "            'compStage': \"\",\n",
    "            'compTag': \"\",\n",
    "            'currentPage': \"0\",\n",
    "            'dq': \"050020\",\n",
    "            'eduLevel': \"\",\n",
    "            'industry': \"\",\n",
    "            'jobKind': \"\",\n",
    "            'key': \"产品经理\",\n",
    "            'pageSize': 40,\n",
    "            'pubTime': \"\",\n",
    "            'salary': \"\",\n",
    "            'suggestTag': \"\",\n",
    "            'workYearCode': \"0\"\n",
    "        },\n",
    "        'passThroughForm':{\n",
    "              \"scene\": \"condition\",\n",
    "              \"skId\": \"8e4q77hx6vf31cg792dkek1g1urespdb\",\n",
    "              \"fkId\": \"8e4q77hx6vf31cg792dkek1g1urespdb\",\n",
    "              \"ckId\": \"2genfj3xapvxmyzzmi8ywhx9idwwpgeb\",\n",
    "              \"suggest\": 'null'\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "headers={\n",
    "    'Accept': 'application/json, text/plain, */*',\n",
    "    'Accept-Encoding':'gzip,deflate,br',\n",
    "    'Accept-Language': 'zh-CN,zh;q=0.9',\n",
    "    'Connection': 'keep-alive',\n",
    "    'Content-Length': '488',\n",
    "    'Content-Type': 'application/json;charset=UTF-8;',\n",
    "    'Cookie':'__uuid=1591862079879.09; __s_bid=a84828c5d83c46720b67bcac8251d5f77f4d; __gc_id=244d297d2a3347d4965b02dba39a7226; _ga=GA1.1.1460161031.1686039917; XSRF-TOKEN=u4tVU_aiQma5X88lOWZ5AA; __tlog=1686041968645.25%7C00000000%7C00000000%7Cs_o_007%7Cs_o_007; acw_tc=276077bd16860419700746034efce4b5417f8f22caa63168b1f9ac6776bef1; Hm_lvt_a2647413544f5a04f00da7eee0d5e200=1686039919,1686041969; Hm_lpvt_a2647413544f5a04f00da7eee0d5e200=1686042095; __session_seq=3; __uv_seq=8; _ga_54YTJKWN86=GS1.1.1686039917.1.1.1686042106.0.0.0',\n",
    "    'Host': 'api-c.liepin.com',\n",
    "    'Origin': 'https://www.liepin.com',\n",
    "    'Referer':'https://www.liepin.com/',\n",
    "    'Sec-Ch-Ua':'\"Not.A/Brand\";v=\"8\", \"Chromium\";v=\"114\", \"Google Chrome\";v=\"114\"',\n",
    "    'Sec-Ch-Ua-Mobile': '?0',\n",
    "    'Sec-Ch-Ua-Platform':'\"Windows\"',\n",
    "    'Sec-Fetch-Dest': 'empty',\n",
    "    'Sec-Fetch-Mode': 'cors',\n",
    "    'Sec-Fetch-Site': 'same-site',\n",
    "    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',\n",
    "    'X-Client-Type': 'web',\n",
    "    'X-Fscp-Bi-Stat':'{\"location\": \"https://www.liepin.com/zhaopin/?city=050020&dq=050020&pubTime=&currentPage=0&pageSize=40&key=%E4%BA%A7%E5%93%81%E7%BB%8F%E7%90%86&suggestTag=&workYearCode=0&compId=&compName=&compTag=&industry=&salary=&jobKind=&compScale=&compKind=&compStage=&eduLevel=&otherCity=&ckId=8e4q77hx6vf31cg792dkek1g1urespdb&scene=condition&skId=8e4q77hx6vf31cg792dkek1g1urespdb&fkId=8e4q77hx6vf31cg792dkek1g1urespdb&sfrom=search_job_pc&suggestId=\"}',\n",
    "    'X-Fscp-Fe-Version':'',\n",
    "    'X-Fscp-Std-Info': '{\"client_id\": \"40108\"}',\n",
    "    'X-Fscp-Trace-Id': 'fb0bce80-b558-4303-b00e-79bc88fe5b18',\n",
    "    'X-Fscp-Version':'1.1',\n",
    "    'X-Requested-With': 'XMLHttpRequest',\n",
    "    'X-Xsrf-Token': 'u4tVU_aiQma5X88lOWZ5AA',\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'flag': 0, 'code': '-1', 'msg': '系统错误，请稍后再试！', 'data': {}}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = session.post(url, data=json.dumps(payload), headers=headers)\n",
    "\n",
    "# extract the JSON data from the response\n",
    "response_data = r.json()\n",
    "\n",
    "# # example: print the number of job postings returned\n",
    "# print(response_data)\n",
    "response_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
